System, Method, and Multi-Level Object Data Structure Thereof For Browsing Multimedia Data

ABSTRACT

System, method, and multi-level object data structure thereof for browsing a multi-media data, the method for browsing a multimedia data using a multilevel object data structure including the steps of (1) receiving multimedia information and multiple supplementary information on each object in the multimedia information on the same time, (2) separating the multimedia information and the multiple supplementary information, (3) displaying the multimedia information, and (4) browsing/searching supplementary information related to a particular object from the multiple supplementary information, and displaying the supplementary information when a user requests for browsing/searching the supplementary information related to the particular object.

CROSS-REFERENCE TO RELATED APPLICATION AND CLAIM OF PRIORITY

The present application is a continuation of co-pending U.S. patentapplication Ser. No. 11/192,731, filed Jul. 29, 2005, entitled “System,Method, and Multi-Level Object Data Structure Thereof for BrowsingMultimedia,” which is a continuation of U.S. patent application Ser. No.09/594,808, filed on Jun. 16, 2000, entitled “System, Method, andMulti-Level Object Data Structure Thereof for Browsing Multimedia,”which claims priority to and the benefit of Korean Patent Application26763/1999, filed on Jul. 3, 1999, entitled “System, Method, andMulti-Level Object Data Structure Thereof for Browsing Multimedia,”which is assigned to the present assignee and is incorporated in itsentirety herein by reference.

BACKGROUND

1. Field of Invention

The present invention relates to a browsing of multi-media data, such asa moving picture, and more particularly, to system, method, andmulti-level object data structure thereof for browsing a multi-mediadata, which permits to browse detailed supplementary information on eachobject included in the multi-media data in displaying the multi-mediadata on request from a user.

2. Background of the Related Art

Recently, as searching techniques for digital moving pictures aredeveloped, there have been active suggestions on browsing or searchingcontents based moving picture. As typical examples, there are browsingby using a key frame disclosed in the U.S. Pat. No. 5,708,767, or anobject based browsing method disclosed in the U.S. Pat. No. 5,655,117.In the browsing by using a key frame, a few typical important scenes/keyframes are suggested, and reproduced in an entire moving picture, tosearch the moving picture. However, since this method is conductedmerely relying on the key frames, it is difficult to expect a searchaccording to actual development of contents of the moving picture. Onthe other hand, in the object based moving picture searching, the movingpicture is depicted and searched with reference to entrance and exit ofdifference objects, such as characters, in the moving picture. However,most of the objects defined in the object based moving picture searchingis foreground objects, of which structures are not appropriate forexpressing place information which is background objects. Therefore, thebackground information, such as a place, can not be used in the relatedart, because, though a portion a specific object enters on the picturemay be defined as an actual entry section in the case of foregroundobjects, the place is defined as one entry section since the place isthe same as far as one scene is continued even if a camera angle ischanged. Therefore, the place has implication different from a simpleentry section in other foreground object as an implication that theplaces are the same is reflected to an object entry section even ifappearances of an object on the picture are different. And, even thougha foreground information with a motion is information only on movementof the object, place information with a motion implies a movement of thecamera. Such place information becomes more useful when the placeinformation is managed as a separate object. However, the watcher maywant supplementary information on an actual place, i.e., where is theplace the picture is taken, or other supplementary information on theactual place. For example, though a café where the heroine drops in isplace information of the moving picture contents, the watcher may wantsupplementary information, such as the place the café actually is, or anintroduction on an address or place (or a sketch map). Or the watchermay want detailed supplementary information on a particular object inthe moving picture in view of contents of the moving picture, or actualsupplementary information (for example, name, brief profile, andinformation on a home page, an address, or fan club, for fans). However,the related art method has a problem in that a searching or browsingmethod which can meet the aforementioned users wants can not beprovided.

SUMMARY OF THE INVENTION

Accordingly, the present invention is directed to system, method, andmulti-level object data structure thereof for browsing a multi-mediadata that substantially obviates one or more of the problems due tolimitations and disadvantages of the related art.

An object of the present invention is to provide system, and method forbrowsing a multi-media data, which permits to browse or search forsupplementary information on a particular object included in a movingpicture based on three elements of object, event, and background duringthe watcher watches the moving picture.

Other object of the present invention is to provide a multi-level objectdata structure for implementing the system, and method for browsing amulti-media data of the present invention.

Additional features and advantages of the invention will be set forth inthe description which follows, and in part will be apparent from thedescription, or may be learned by practice of the invention. Theobjectives and other advantages of the invention will be realized andattained by the structure particularly pointed out in the writtendescription and claims hereof as well as the appended drawings.

To achieve these and other advantages and in accordance with the purposeof the present invention, as embodied and broadly described, the methodfor browsing a multimedia data using a multilevel object data structure,including the steps of (1) receiving multimedia information and multiplesupplementary information on each object in the multimedia informationon the same time, (2) separating the multimedia information and themultiple supplementary information, (3) displaying the multimediainformation, and (4) browsing/searching supplementary informationrelated to a particular object from the multiple supplementaryinformation, and displaying the supplementary information when a userrequests for browsing/searching the supplementary information related tothe particular object.

The object related supplementary information includes contentinformation on a particular program in the multimedia information, andmultiple supplementary information including real information.

The content information and the real information includes selectivecombinations of information which represents relations of objectinformation, event information, place information, andobject/event/place information in a graph.

The content information and the real information includes textinformation describing character of the object, and image or graphicinformation on the character.

The object information includes text information having informationdescribing the object, and image information on the object information.

The event information includes text information describing contents ofthe event, and information on a place or a sketch map the eventoccurred.

The place information includes text information describing the place,and information on location or a sketch map of the place.

In other aspect of the present invention, there is provided amulti-level object data structure in a system for displaying multimediainformation, wherein supplementary information on each object includedin the multimedia information has multiple supplementary informationstructure including at least content information and real information ona particular program.

In another aspect of the present invention, there is provided amulti-level object data structure wherein contents of a moving pictureis expressed in a graph of object and place relation and an array ofrelated events are made for each relation for making relation, and, forproviding a function for displaying a required section, the movingpicture is expressed in an object structure for providing information onobjects, a place structure for providing information on a place which isa background of an event, an event structure for providing informationrequired for describing a particular unit event in the moving picture,and relations of the structures in a graph, for searching the movingpicture by using a double structure of content information and realinformation on at least one relation for each of the relations or aselected element, in the moving picture.

The object structure, the place structure, and the event structure,expressed as content information and real information of the movingpicture, are expressed in text information describing at least objectname, place name, and event.

The content information on the moving picture is information on a placeor an object having a meaning in view of content of the moving picture,and the real information is information on a real place which ismeaningful place in view of the content of the moving picture, or onactual performer cast an object in the moving picture.

In further aspect of the present invention, there is provided a systemfor browsing a moving picture by using a multi-level object datastructure including means for supplying moving picture digitalinformation including a multiple object information structure havingcontent based meaningful object information of the moving picture andreal object information of the object, means for receiving the digitalinformation and separating and reading the multiple object informationand the moving picture information, and means for presenting real objectinformation the multiple object information describes in response to ausers request.

The multiple object information structure has multiple objectinformation structures for the object, the place, and the event, and themeans for presenting real object information presents real informationon an object cast in the moving picture or real place information on theplace which is a background of an event.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory and areintended to provide further explanation of the invention as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are included to provide a furtherunderstanding of the invention and are incorporated in and constitute apart of this specification, illustrate embodiments of the invention andtogether with the description serve to explain the principles of theinvention:

In the drawings:

FIG. 1 illustrates one frame of a moving picture for explaining thepresent invention;

FIG. 2 illustrates a state of browsing of place information on aparticular object when a moving picture is displayed in accordance witha preferred embodiment of the present invention;

FIG. 3 illustrates a block diagram of a display for browsing aparticular object in a moving picture in accordance with anotherpreferred embodiment of the present invention;

FIG. 4 illustrates a place DS of a particular object when a movingpicture is displayed in accordance with a preferred embodiment of thepresent invention;

FIG. 5 illustrates a semantic DS showing relation with object/placeincluded in a moving picture in a form of a graph in searching/browsingthe moving picture in accordance with a preferred embodiment of thepresent invention;

FIG. 6 illustrates a detail of the event DS in FIG. 5;

FIG. 7 illustrates a relation between object and place included in amoving picture in a form of object/place/event relation graph indisplaying a moving picture in accordance with a preferred embodiment ofthe present invention;

FIG. 8 illustrates multiple object DS included in a moving picture inaccordance with a preferred embodiment of the present invention;

FIG. 9 illustrates an character DS in displaying a moving picture inaccordance with a preferred embodiment of the present invention; and,

FIG. 10 illustrates a state of browsing of a particular object includedin a moving picture by using multiple object DS in accordance with apreferred embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT

Reference will now be made in detail to the preferred embodiments of thepresent invention, examples of which are illustrated in the accompanyingdrawings. A transmission side, such as a VOD system or a digital TVbroadcasting station transmits a particular video signal to a display ina receiver side according to a request from a user. In this instance,the video signal is transmitted, together with coded multipleinformation having detailed supplementary information on each objectincluded in the video signal. Therefore, the display in the receiverside receives all the video signal and the detailed supplementaryinformation from the transmission side, and displays the video signalonly. FIG. 1 shows one scene of a drama with a hospital in a background.During watching a video program a receiver side display displays, thewatcher may request for browsing detailed supplementary information onan object, such as the hospital by handling a controller, such as aremote controller. The detailed supplementary information may be programcontent based information of which video signal is provided, or actualinformation. If the watcher in the receiver side requests for browsingof actual information on the hospital shown in FIG. 1, the receiver sidebrowsers and displays a name 201, an address 202, and description of thehospital 203, and a sketch map 204 of the actual hospital as shown inFIG. 2. On the other hand, the watcher may request for browsing ofcontent information on the hospital in the picture in FIG. 1 for betterunderstanding of the story of the video program. In this instance, thereceiver side display may browse and display the detailed information onthe content basis hospital according to such a users request. In FIG. 2,if a network environment is supported for the receiver side display suchthat the receiver side display can be connected to the internet, thepresent frame on the display may be linked, and shift to an internetaddress (URL) where there is more detailed information on the place. InFIGS. 1 and 2, though the user requests for browsing content basedinformation or actual information on a place, like the hospital, byhandling the remote controller in the middle of watching the videoprogram, the same method may be applicable to an object or an event.That is, actual information on a content based specific object in themoving picture program may be requested. The object may be extended evento an animal or virtual character in the moving picture program. Forexample, necessary detailed supplementary information on a performer theuser likes, among performers in a drama, such as information on a name,profile, a fan club, an internet home page address, and the like, may berequested.

FIG. 3 illustrates a block diagram of a display for browsing aparticular object in a moving picture in accordance with anotherpreferred embodiment of the present invention.

That is, FIG. 3 is an example for serving actual object relatedinformation as a coded data based on multiple information structureother than the moving picture, wherein the VOD server 301 and theinternet server 302 shown in FIG. 3 are connected to a communicationnetwork 309, and the communication network 309 and a digital TVbroadcasting system 303 are connected to a receiver 305 of the receiverside display 304. In FIG. 3, the receiver 305 in the receiver sidedisplay 304 receives a video signal from the communication network 309or the broadcasting system 303, and a decoder 306 decodes the videosignal, to separate an original moving picture data and a supplementaryinformation data. Then, the separated original video data is displayedon the display directly, while the supplementary information data isstored in a memory 307, for using on request for browsing of contentbased supplementary information or actual supplementary information on aparticular object in the moving picture. In FIG. 3, the control unit 308controls browsing operation for the detailed supplementary informationof the object in response to the request of the watcher. In thisinstance, if the receiver side display 304 is supported such that thereceiver side display 304 is interlocked with a VOD system server or aninternet server, the supplementary information of each object includedin the video signal is not encoded with an original moving picture data,but is managed at the VOD server or the internet server separately, andprovided at the watchers request. In the present invention, there is nospecial limitation on a position and a searching method of each objectincluded in the video signal, and when and how the browsing isconducted. Therefore, as explained, the method for displaying thedetailed supplementary information of each object included in the videosignal will bring about a new advertising effect. For example, if aparticular café, restaurant, or the like provides the detailedsupplementary information on the café, restaurant, or the like as thecafé, restaurant, or the like support the place in production of themoving picture program, the watcher can obtain supplementary information(for example, place, address, description, and sketch map and etc.,) onthe actual supported place in the moving picture when necessary.

FIG. 4 illustrates a place DS of a particular object when a movingpicture is displayed in accordance with a preferred embodiment of thepresent invention. Referring to FIG. 4, the place DS 401 of the presentinvention includes a real location 402 and a content location in amoving picture. The content location 403 includes an annotation 404, adescription of a name, or location of the place in a form of a text.And, the real location 402 includes a geometrical info 405 and anannotation 406, a description of a name, or location of the real placein a form of a text. Therefore, the content location information is usedfor searching a moving picture by using object/place relation graph, andthe real place information is used in providing information on the realplace at users request.

FIG. 5 illustrates a semantic DS showing relation with object/placeincluded in a moving picture in a form of a graph in searching/browsingthe moving picture in accordance with a preferred embodiment of thepresent invention. Referring to FIG. 5, the semantic DS 501 of thepresent invention includes an event DS 502, an object DS 503, a place DS504, an event/object/place relation DS 505 in which event/object/placerelation are shown in a graph. The object DS, the place DS, and theevent DS includes a text information for explaining the object, theplace, and the event, respectively.

FIG. 6 illustrates a detail of the event DS in FIG. 5. Referring to FIG.6, the event DS 601 of the present invention includes a reference tosegment 602 and an annotation DS. Therefore, the receiver side displaycan display a relevant moving picture section by using such an event DS.That is, while text information explaining an event is provided, amoving picture section corresponding to the event is displayed by usingthe segment information. And, in FIG. 6, one event may include lowerlevel event information explained in detail relatively. By using theevent DS of such a hierarchical structure, object/place relation graphcan be applied, not to an entire section of the moving picture, but to aportion of the section.

FIG. 7 illustrates a relation between object and place included in amoving picture in a form of object/place/event relation graph indisplaying a moving picture in accordance with a preferred embodiment ofthe present invention. Referring to FIG. 7, the object/place/eventrelation DS 701 of the present invention may be represented in an entityrelation 702, inclusive of a reference to place 703, a reference toobject 704, and a reference to event 705. In the object/place relation,one or more events may be related to one another, or only object andevent are related, or place and event are related.

FIG. 8 illustrates multiple object DS included in a moving picture inaccordance with a preferred embodiment of the present invention.Referring to FIG. 8, the multiple object DS 801 of the present inventionincludes a real object 802 and a content object 803 in a moving picture.The content object 803 includes an annotation which explains an objectname or other information in a form of text, and the real object 802includes an annotation 805 which explains the real object in a form oftext. Thus, the content object is used for understanding a content ofthe moving picture, such as a moving picture searching by using anobject/place (event when extended) relation graph, and the real objectinformation is used for browsing or searching for the real object at ausers request.

FIG. 9 illustrates a multiple object DS in FIG. 8 in accordance withanother preferred embodiment of the present invention, showing amultiple character IDS. Referring to FIG. 9, the character DS 901 of thepresent invention includes a real character 902 and a content character903. The content character 903 includes an annotation 904 which explainscharacter information created according to name, character, and storydevelopment of the object in a content of the moving picture in a formof a text, and the real character 902 includes actor info of an actualparticular object, and an annotation 906 explaining a name, a profile,the most important work and the like of the real character in a form ofa text. The content character information is used for understanding acontent of the moving picture, such as moving picture searching by usingobject/place relation graph, and real character is used inbrowsing/searching a real character (actor/actress) at users request.

FIG. 10 illustrates a state of browsing of a particular object includedin a moving picture by using multiple object DS in accordance with apreferred embodiment of the present invention. In FIG. 10, 101 expressesan outline of a video story by using the most schematic event structure,and 102 represents object-place relations set up within an event sectionselection in the outline 101. And, 103 explains detailed events meetingrelations (human B-hospital) selected on the object-place relation graph102.

Though multiple object DS of multiple character DS or multiple place DSis explained in the aforementioned embodiment of the present invention,multiple even DS can also be explained in the same method. For example,when contents of a moving picture is composed based on an actual event(or character), event information in a moving picture content and anactual event information may be provided on the same time.

As has been explained, because each object included in a video signalhas content basis supplementary information on a moving picture andactual supplementary information on the same time when a moving picturesignal is displayed on a display, the watcher may use the supplementaryinformation in browsing/searching the moving picture for understandingcontent of the moving picture when necessary, as well as the actualsupplementary information, thereby providing a new concept ofadvertising.

And, by providing object, place, and event structures, and a semanticstructure representing relation between the structures, an movingpicture browsing/searching can be made effectively based on the threeelements when the moving picture is displayed. And, in order to carryout effective display of multimedia data of the present invention,particular object data structure is provided.

It will be apparent to those skilled in the art that variousmodifications and variations can be made in system, method, andmulti-level object data structure thereof for browsing a multi-mediadata of the present invention without departing from the spirit or scopeof the invention. Thus, it is intended that the present invention coverthe modifications and variations of this invention provided they comewithin the scope of the appended claims and their equivalents.

1. A method implemented by an apparatus for using multimedia data, theapparatus including a processor, a storage device and a display, themethod comprising: storing, in the storage device, metadata describingat least a portion of a video program, the described portion of thevideo program including at least one segment depicting a first characterplayed by a first actor, said metadata having a multi-level hierarchicalstructure and including character list data describing personsassociated with said video program, wherein said character list dataallows content referencing based on user input, and wherein thecharacter list data includes a first character list item correspondingto the first character depicted in the video program, the firstcharacter list item including a character name to identify the firstcharacter and an actor name to identify the first actor playing thefirst character in the video program; and using the processor insearching for said at least one segment depicting said first characterbased on said metadata to playback said at least one segment on thedisplay.
 2. The method of claim 1, further comprising: receiving said atleast one segment of the video program; and playing back said receivedsegment on the display.
 3. The method of claim 2, wherein receiving saidat least one segment of the video program includes receiving the entirevideo program from a digital broadcasting system.
 4. The method of claim1, wherein searching for said at least one segment depicting said firstcharacter based on said metadata includes using information in the firstcharacter list item.
 5. The method of claim 1, wherein storing themetadata includes receiving the metadata from a digital broadcastingsystem.
 6. The method of claim 1, wherein the first character list itemis a container in the hierarchical structure and includes a characterelement and an actor element, the character element including thecharacter name and the actor element including the actor name.
 7. Themethod of claim 6, wherein the actor element further includes additionalinformation about the first actor.
 8. An apparatus for using multimediadata, the apparatus comprising: a display; a storage device; and aprocessor configured to: store, in the storage device, metadatadescribing at least a portion of a video program, the described portionof the video program including at least one segment depicting a firstcharacter played by a first actor, said metadata having a multi-levelhierarchical structure and including character list data describingpersons associated with said video program, wherein said character listdata allows content referencing based on user input, and wherein thecharacter list data includes a first character list item correspondingto the first character depicted in the video program, the firstcharacter list item including a character name to identify the firstcharacter and an actor name to identify the first actor playing thefirst character in the video program; and search for said at least onesegment depicting said first character based on said metadata toplayback said at least one segment on the display.
 9. The system ofclaim 8, wherein the processor is further configured to: receive said atleast one segment of the video program; and play back said receivedsegment on the display.
 10. The system of claim 9, wherein receivingsaid at least one segment of the video program includes receiving theentire video program from a digital broadcasting system.
 11. The systemof claim 8, wherein searching for said at least one segment depictingsaid first character based on said metadata includes using informationin the first character list item.
 12. The system of claim 8, whereinstoring the metadata includes receiving the metadata from a digitalbroadcasting system.
 13. The system of claim 8, wherein the firstcharacter list item is a container in the hierarchical structure andincludes a character element and an actor element, the character elementincluding the character name and the actor element including the actorname.
 14. The system of claim 13, wherein the actor element furtherincludes additional information about the first actor.